Adjustable camera mount for a vehicle windshield

ABSTRACT

A camera mount for mounting a camera inside a windshield of a vehicle. The camera includes a lens mount and a camera housing. The front tip of the lens mount is constrained to be in close proximity to or constrained to contact the inside of the windshield for different rake angles of the windshield.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority from U.S. patent application,61/193,528 filed by the present inventor on 5 Dec. 2008, and from U.S.patent application Ser. No. 12/573,836 filed by the present inventor on5 Oct. 2009, the disclosures of which are incorporated herein byreference.

BACKGROUND

1. Technical Field

The present invention relates to multiple driver assistance systems andmore specifically to the integration of the driver assistance systemsonto a single hardware platform.

2. Description of Related Art

During the last few years camera based driver assistance systems (DAS)have been entering the market; including lane departure warning (LDW),Automatic High-beam Control (AHC), traffic sign recognition (TSR) andforward collision warning (FCW).

Lane departure warning (LDW) systems are designed to give a warning inthe case of unintentional lane departure. The warning is given when thevehicle crosses or is about to cross the lane marker. Driver intentionis determined based on use of turn signals, change in steering wheelangle, vehicle speed and brake activation. There are various LDW systemsavailable. One algorithm for lane departure warning (LDW) used by theassignee (Mobileye Technologies Ltd., Nicosia, Cyprus, hereinafter“Mobileye”) of the present application is predictive in that it computestime to lane crossing (TLC) based on change in wheel-to-lane distanceand warns when the time-to-lane crossing (TLC) is below a certainthreshold. Other algorithms give a warning if the wheel is inside acertain zone around the lane marker. In either case, essential to thelane departure warning system is the lane marker detection algorithm.Typically, the lane markers are detected in the camera image and then,given the known camera geometry and camera location relative to thevehicle, the position of the vehicle relative to the lane is computed.The lane markers detected in the camera image are then collected overtime, for instance using a Kalman filter. Wheel-to-lane marker distancemay be given with an accuracy of better than 5 centimeters. With aforward looking camera, wheel-to-lane marker distance is not observeddirectly but is extrapolated from the forward view of the camera. Thecloser road markings are observed, less extrapolation is required fordetermining wheel-to-lane marker distance and more accurate estimates ofwheel-to-lane marker distance are achieved especially on curves of theroad. Due to the car hood and the location of the camera, the road isseldom visible closer than six meters in front of the wheels of the car.In some cars with longer hoods, minimal distance to visible road infront of the car is even greater. Typically the lane departure warningsystem of Mobileye works on sharp curves (with radius down to 125 m).With a horizontal field of view (FOV) of 39 degrees of the camera, theinner lane markers are still visible on curves with a radius down to 125meters. In order to correctly perform lane assignment on curves, lanemarkings are detected at 50 meters and beyond. With a horizontal fieldof view (FOV) of 39 degrees for the camera, a lane mark of width 0.1meters at 50 m distance corresponds in the image plane to just under twopixels wide and can be detected accurately. The expectation from thelane departure warning systems is greater than 99% availability whenlane markings are visible. Expectation with 99% availability isparticularly challenging to achieve in low light conditions when thelane markings are not freshly painted (have low contrast with the road)and the only light source is the car halogen headlights. In low lightconditions, the lane markings are only visible using the highersensitivity of the clear pixels (i.e. using a monochrome sensor or ared/clear sensor). With the more powerful xenon high intensity discharge(HID) headlights it is possible to use a standard red green blue (RGB)sensor in most low light conditions.

The core technology behind forward collision warning (FCW) systems andheadway distance monitoring is vehicle detection. Assume that reliabledetection of vehicles in a single image a typical forward collisionwarning (FCW) system requires that a vehicle image be 13 pixels wide,then for a car of width 1.6 m, a typical camera (640×480 resolution and40 deg FOV) gives initial detection at 115 m and multi-frame approval at100 m. A narrower horizontal field of view (FOV) for the camera gives agreater detection range however; the narrower horizontal field of view(FOV) will reduce the ability to detect passing and cutting-in vehicles.A horizontal field of view (FOV) of around 40 degrees was found byMobileye to be almost optimal (in road tests conducted with a camera)given the image sensor resolution and dimensions. A key component of atypical forward collision warning (FCW) algorithm is the estimation ofdistance from a single camera and the estimation of scale change fromthe time-to-contact/collision (TTC) as disclosed for example in U.S.Pat. No. 7,113,867.

Traffic sign recognition (TSR) modules are designed typically to detectspeed limit signs and end-of-speed limit signs on highways, countryroads and urban settings. Partially occluded, slightly twisted androtated traffic signs are preferably detected. Systems implementingtraffic sign recognition (TSR) may or should ignore the following signs:signs on truck/buses, exit road numbers, minimum speed signs, andembedded signs. A traffic sign recognition (TSR) module which focuses onspeed limit signs does not have a specific detection range requirementbecause speed limit signs only need to be detected before they leave theimage. An example of a difficult traffic sign to detect is a 0.8 meterdiameter traffic sign on the side of the road when the vehicle isdriving in the center lane of a three lane highway. Further details of aTSR system is disclosed by the present assignee in patent applicationpublication US20080137908.

A typical automatic headlight or high/low beam control (AHC) systemdetects the following conditions and switches from high beams to lowbeams: headlights of oncoming vehicles, taillights of precedingvehicles, street lights or ambient light indicating that high beams arenot required and a low vehicle speed. The host vehicle lights areswitched back to high beams when none of these conditions exist (oftenafter a specified grace period). One approach for detecting taillightsis to compare images from two sensors: one with a red filter and thesecond with a cyan filter. The cyan filter responds to non-red lightsources and will give zero response to red light. By comparingcorresponding pixels from two imaging sensors one can detect the colorof the light source. The number of pixels of each color above a certainintensity is counted and if the count is above a threshold the systemsswitches to low beams. The use of color filters with imaging sensors maypreclude the simultaneous use of the same image frames for other driverassistance applications.

A second approach for automatic high-beam control (AHC) uses an RGBsensor to give better color differentiation. Typical light sources canbe located in the full CIE color space as defined by the InternationalCommission on Illumination. This approach distinguishes between green,yellow and red lights. A powerful green traffic light is not confusedwith an oncoming vehicle. Since a single sensor with a color mosaicfilter i.e. Bayer pattern mosaic is used, the lens is defocused so as tospread a light source over multiple pixels. The use of the color mosaicfilter reduces both the effective image sensor resolution (by 50%) andthe intensity response (to less than one third). The color mosaic filtermay preclude the use of the same sensor for traffic sign recognition(TSR) or lane departure warning (LDW) because of the intensity responsepenalty.

Given that forward collision warning (FCW), traffic sign recognition(TSR) and lane departure warning (LDW) already require a high resolutionmonochrome sensor, a new automatic high-beam control (AHC) algorithm wasdeveloped for use with high resolution monochrome sensors as disclosedin U.S. Pat. No. 7,566,851. A number of different pattern recognitiontechniques are used with higher resolution monochrome imaging sensors toidentify light sources instead of relying on color information. Theautomatic high-beam control (AHC) algorithm includes the followingfeatures: Detect bright spots in the sub-sampled long exposure image andthen perform clustering and classification in the full resolution image,classify spots based on brightness, edge shape, internal texture, getfurther brightness information from the short exposure frames andclassify obvious oncoming headlights based on size and brightness, trackspots over time and compute change in size and brightness, pair upmatching spots based on similarity of shape, brightness and motion,classify pairs as oncoming or taillights based on distance, brightnessand color, and estimate distance and where unmatched spots might bemotorcycles taillights.

The term “electronic traffic sign” as used herein refers to variablemessage signs, pulsed electronic traffic signs and/or back-lit trafficsigns. A variable message sign (VMS) and in the UK known as a matrixsign is an electronic traffic sign often used on roadways to givetravelers information about special events. Such signs warn of trafficcongestion, accidents, incidents, roadwork zones, or speed limits on aspecific highway segment. In urban areas, VMS are used within parkingguidance and information systems to guide drivers to available carparking spaces. They may also ask vehicles to take alternative routes,limit travel speed, warn of duration and location of the incidents orjust inform of the traffic conditions

Pulsed electronic traffic signs include an array on light emittingdiodes (LEDs) Typically the light emitting diodes (LEDs) in electronicsigns are not on all the time but are pulsed at a high frequency(typically 80 Hz to 160 Hz) giving varying cycle times for the lightemitting diodes (LEDs) in electronic signs (typically between 12.5 mS to5.25 mS). The LED frequency, LED duty cycle and LED light intensity varyfrom electronic sign to electronic sign. Moreover, electronic signs arenot uniform across all countries and even vary according to the time ofday and ambient light levels. However, the typical cycle time of all LEDelectronic signs is smaller than 11.4 milliseconds and longer than 6milliseconds.

Support vector machines (SVMs) are a set of related supervised learningmethods used for classification and regression. Support vector machines(SVMs) belong to a family of generalized linear classifiers Supportvector machines (SVMs) can also be considered a special case of Tikhonovregularization. A special property of SVMs is that they simultaneouslyminimize the empirical classification error and maximize the geometricmargin; hence they are also known as maximum margin classifiers.

Viewing the input data as two sets of vectors in an n-dimensional space,an SVM will construct a separating hyper-plane in that space, one whichmaximizes the “margin” between the two data sets. To calculate themargin, two parallel hyper-planes are constructed, one on each side ofthe separating one, which are “pushed up against” the two data sets.Intuitively, a good separation is achieved by the hyper-plane that hasthe largest distance to the neighboring data points of both classes. Thehope is that, the larger the margin or distance between these parallelhyper-planes, the better the generalization error of the classifier willbe.

In geometry, a two-dimensional Bravais lattice, studied by AugusteBravais (1850), is a set of points generated by a set of discretetranslation operations described by:R=n ₁ ā ₁ +n ₂ ā ₂where n_(i) are any integers and ā_(i) are known as the primitivevectors which lie in a plane and span the lattice. For any choice ofposition vector, the lattice looks the same.

The term “exposure” and “exposure time” are used herein interchangeablyand refers to the time duration of image integration in an image sensor.

The term “detection” in the context of traffic sign detection as usedhereinafter refers to detecting that there is a putative traffic sign inthe environment of the vehicle such as by detecting the outline of atraffic sign. The term “recognition” in the context of traffic signrecognition refers to reading and/or interpreting the content or meaningof the traffic sign.

The terms “maximal” and “minimal” in the context of transmittance ofoptical filters are relative terms, for instance a range of wavelengthsin which there is “maximal transmittance means that there is a relativemaximum on the average in that wavelength range compared with theadjacent ranges of wavelengths.

The term “substantial” transmittance refers to transmittance greaterthan eighty percent on the average in the wavelength range. The term“insubstantial” transmittance means less than twenty percenttransmittance on the average in the wavelength range.

A camera has six degrees of freedom, three of which involve rotations:pan, tilt and roll. A “pan” is a rotary pivoting movement of the camerafrom left to right or vice versa. In a three dimensional coordinatesystem where y is the vertical axis, a “pan” is a rotation of the cameraaround the vertical or y axis. A “tilt” is an up and/or down movement ofthe camera, or a rotation around the horizontal or x axis. A “roll” is arotation of the camera about the z axis that is the optical axis of thecamera. These terms are analogous to the aeronautical terms yaw, pitchand roll. Yaw is synonymous with pan. Pitch is synonymous with tilt, androll has the same meaning in both nomenclatures.

The term “camera parameter” as used herein in the context of calibrationof a camera refers to one or more of the six degrees of freedom, forinstance camera height from the ground and/or camera orientationparameters, e.g. “pitch”, “roll” and “yaw”.

The term “rake angle” of a windshield of a vehicle is the angle betweenthe vertical and the surface of the windshield in the middle of thewindshield where the windshield approximates a plane surface.

The term “partitioning” or “to partition” as used herein refers toassigning different attributes to different image frames, for example bycapturing different partitions with different camera parameters, e.g.gain, exposure time. The term “partitioning” as used herein does notrefer to dividing an image frame into parts, e.g. two halves of of animage frame. In the context of “image frames”, the terms “portion” and“partition” are used herein interchangeably.

BRIEF SUMMARY

According to an embodiment of the present invention there is provided amethod for performing a driver assistance function using a computerizedsystem mounted on a moving vehicle. The computerized system includes acamera and an image processor. Under control of the image processor thecamera is adapted for capturing in real time image frames of theenvironment in the field of view of the camera. A first portion and asecond portion of the image frames are partitioned. For the firstportion, a camera parameter is controlled based on ambient lightingconditions in the environment in the field of view of the camera. Basedon the camera parameter of the first portion, a camera parameter of thesecond portion is set. The image frames are captured and transferred tothe image processor for processing of the driver assistance function.The camera parameters of the first and second portions are gain and/orexposure time. When the driver assistance function includes detectionand recognition of traffic signs, the detection of the traffic signsuses the first portion of the image frames and the recognition of thetraffic sign uses the second portion of the image frames. When thetraffic sign is an electronic traffic sign, both the first portion ofthe image frames and the second portion of the image frames may be usedfor recognition of the traffic sign. The electronic traffic sign may bea pulsed electronic traffic sign, a variable message traffic sign and/ora back-lit traffic sign. For the second portion of the image frames, ashort exposure time is set which is substantially shorter than theexposure time of the first portion. The camera parameter of the secondportion of the image frames is set substantially differently betweenday-mode operation during the day and night-mode operation during thenight. Optionally the camera parameter of the second portion is setsubstantially differently during dusk for dusk-mode operation thanduring the day-mode operation and/or the camera parameter of the secondportion is set substantially differently during dusk for dusk-modeoperation than during the night-mode operation.

A third portion of the image frames is optionally partitioned and forthe third portion a yet shorter exposure time is set substantiallyshorter than the short exposure time of the second portion. When thetraffic sign is an electronic traffic sign, during the day therecognition of the electronic traffic sign may be performed using allthree portions of the image frames each with the substantially differentexposure times and at night, the third portion is typically used forautomatic headlight control. At night, exposure of the second portion ofthe image frames is optionally toggled between values selected to becomplementary to the camera parameters of image frames of the thirdportion. When the driver assistance function includes detection oftraffic signs, a gain is controlled based on a portion of the imageframes including an image of the road and not on an a portion of theimage frames image where traffic signs are likely to be imaged.

According to an embodiment of the present invention there is provided atraffic sign recognition system including a detection mechanism adaptedfor detecting a candidate traffic sign and a recognition mechanismadapted for recognizing the candidate traffic sign as being anelectronic traffic sign. A partitioning mechanism may be adapted forpartitioning the image frames into a first partition and a secondpartition. The detection mechanism may use the first portion of theimage frames and the recognition mechanism may use the second portion ofthe image frames. When the candidate traffic sign is detected as anelectronic traffic sign, the recognition mechanism may use both thefirst partition of the image frames and the second portion of the imageframes.

According to an embodiment of the present invention there is provided apatterned color filter including: a yellow (Y) portion adapted formaximal transmittance of light of wavelength between 550 nanometers and800 nanometers and modified cyan (C) portion adapted for maximaltransmittance of light of wavelength between 400 and 550 nanometers andadapted for minimal transmittance for light of wavelengths between 600and 800 nanometers. A magenta (M) portion is optionally adapted forminimal transmittance of visible light of wavelengths between 500 and600 nanometers and maximal transmittance between 400 and 500 nanometersand above 600 nanometers. The yellow portion is substantially equal inarea to the modified cyan portion and the magenta portion combined. Themagenta portion usually includes one quarter of the area of thepatterned filter and the modified cyan portion usually includes onequarter of the area of the patterned filter. The yellow portion, themodified cyan portion and the magenta portion are usually symmetricallydisposed on the patterned color filter as a two-dimensional Bravaislattice including multiple cells. Each of the cells are included in onlyone of the colored portions of the yellow portion, the modified cyanportion and the magenta portion. The nearest neighboring cells of theBravais lattice are included in different colored portions of the yellowportion, the modified cyan portion and the magenta portion. Thetwo-dimensional Bravais lattice may be defined by two primitive vectors.In the direction of one or both of the primitive vectors the coloredportions may be ordered alternately according to YCYCYC . . . and MYMYMY. . . . In the direction of one of the primitive vectors the coloredportions may be ordered according to: YCYMYCYMYCYM . . . . The cells areof dimension corresponding to picture elements of an image sensor.

According to an embodiment of the present invention there is provided adriver assistance system including an image sensor including multiplepicture elements (pixels). An image processor connects to the imagesensor. The image processor is adapted for receiving image frames fromthe image sensor. The image sensor and the image processor are adaptedfor mounting on a vehicle. A patterned color filter filters light priorto sensing by the image sensor. The patterned color filter includes ayellow (Y) portion adapted for maximal transmittance of light ofwavelength between 550 nanometers and 800 nanometers and modified cyan(C) portion adapted for maximal transmittance of light of wavelengthbetween 400 and 550 nanometers and adapted for minimal transmittance forlight of wavelengths above 600 to at least 800 nanometers. A magenta (M)portion is optionally adapted for minimal transmittance of visible lightof wavelengths between 500 and 600 nanometers and maximal transmittancebetween 400 and 500 nanometers and above 600 nanometers. The yellowportion is substantially equal in area to the modified cyan portion andthe magenta portion combined. The magenta portion usually includes onequarter of the area of the patterned filter and the modified cyanportion usually includes one quarter of the area of the patternedfilter. The yellow portion, the modified cyan portion and the magentaportion are usually symmetrically disposed on the patterned color filteras a two-dimensional Bravais lattice including multiple cells. Each ofthe cells are included in only one of the colored portions of the yellowportion, the modified cyan portion and the magenta portion. The nearestneighboring cells of the Bravais lattice are included in differentcolored portions of the yellow portion, the modified cyan portion andthe magenta portion. The two-dimensional Bravais lattice may be definedby two primitive vectors. In the direction of one or both of theprimitive vectors the colored portions may be ordered alternatelyaccording to YCYCYC . . . and MYMYMY . . . . In the direction of one ofthe primitive vectors the colored portions may be ordered according to:YCYMYCYMYCYM . . . wherein in the direction of the second primitivevector the colored portions are ordered alternatively YCYCYC . . . andYMYMYM . . . . Each of the cells are of dimension corresponding tomultiple picture elements (pixels) of the image sensor. The capturedimage frames include three separable images corresponding to the coloredportions. The image processor uses the image from the yellow portion forboth lane departure warning and traffic sign recognition. The imageprocessor optionally uses the image from the yellow portion and othercolor information from one of the images from the cyan and the magentaportions for lane departure warning. The image processor may beconfigured to use at least two of the three separable images to detecttraffic signs. The image processor is configured to use the image of themodified cyan portion to distinguish between headlights and taillights.

According to an embodiment of the present invention there is provided amethod using a computerized system including a camera mounted on a hostvehicle. Under control of an image processor, the camera is adapted forcapturing in real time image frames of the environment in the field ofview of the camera. An image feature is located in a first image frame.The image feature includes an image of a point of the road surfacebeneath the preceding vehicle. The image feature is tracked to a secondimage frame. The time stamp of the second image frame is subtracted fromthe time stamp of the first image frame to produce a time difference.The distance to the preceding vehicle may be computed based on the timedifference and/or a headway warning may be provided based on the timedifference. The headway may be estimated using the time difference. Theheadway is the time required for the host vehicle to reach the currentposition of the preceding vehicle. The image feature may include a pairof image points separated by a distance in image space of the roadsurface. The time-to-contact may be computed based on a difference of adistance in image space between the first image frame and the secondimage frame. Time-to-contact may be computed as the time differencedivided by the relative change of the distance.

According to an embodiment of the present invention there is provided amethod for estimating headway between a host vehicle and a precedingvehicle. The headway is the time required for the host vehicle to reachthe current position of the preceding vehicle. A computerized systemincluding a camera and an image processor is mounted on the hostvehicle. Under control of the processor the camera is adapted forcapturing in real time multiple image frames i of the environment in thefield of view of the camera. An image feature is located in the imageframes. The image feature includes a distance d_(i) in image spaceimaged from a portion of the road surface beneath the preceding vehicle.The image feature is subsequently tracked over the image frames iusually until the image feature reaches the bottom of one of imageframes i. The distance d_(i) in image space is measured for the imageframes i. Time t_(i) is measured based on respective time stamps of theimage frames i. The reciprocal 1/d_(i) of the distance d_(i) is computedas a function of time. The function is fit to a curve on a time axis andthe curve is extrapolated toward zero. The intercept of the function1/d_(i) with the time axis gives an estimate of the headway. When thehost vehicle acceleration is negligible, the curve substantially fits aline. When host vehicle acceleration is substantial the curvesubstantially fits a parabola.

According to an embodiment of the present invention, there is provided amethod for detecting fog in the environment of a moving vehicle. Acomputerized system is mounted on the moving vehicle. The systemincludes a camera and an image processor. Under control of the processorthe camera is adapted for capturing in real time multiple image framesof the environment in the field of view of the camera. A halo isdetected in the image frames and the halo is classified as being causedby light scattered from the fog.

Optionally, a light source is synchronized with the capture of the imageframes so that, a first image frame is captured when the light source isswitched on a second image frame is captured when the light source isswitched off. The second image frame may be subtracted from the firstimage frame or otherwise compared or correlated. The halo is caused bythe light originating from the light source. The light source may be atail light of the vehicle. When the light originates from two taillightsof the vehicle, the halo is symmetrically distributed horizontallywithin the image frames. Alternatively, a headlamp may be modified sothat a portion of the light emitted from the headlamp propagates upwardand the halo is caused by the portion of light propagating upward. Theclassification may be performed by applying at least one rule indicatingthe presence of light scatter above a previously determined threshold.Alternatively, the classification may be performed by previouslytraining a classifier based on known atmospheric conditions of theenvironment in the vicinity of the vehicle to produce a trainedclassifier.

The classification is then performed s by applying the trainedclassifier. The trained classifier may be least one binary classifier ormay be based on support vector machines.

According to an embodiment of the present invention there is provided amethod for calibrating a camera in a vehicle for use in a driverassistance system. The method is adapted for setting camera orientationand/or camera height parameter. An upright target is placed in the fieldof view of the camera at a first distance in front of the vehicle.Typically the first distance is zero—the target is placed touching thefront bumper centered on the bumper using standard features on thevehicle such as the license plate or the car makers emblem. A firstimage is captured a first image of the upright target. The uprighttarget is relocated at a second distance in front of the vehicle and asecond image is captured of the upright target. Mapping may be performedbetween image coordinates of the first image to matching imagecoordinates of the second image. A focus of expansion (FOE) may becomputed. A stationary point of the mapping yields the focus ofexpansion (FOE). When the camera is not laterally centered in thevehicle, the stationary point may be adjusted laterally. Camera heightmay be computed from the vertical image coordinate of the stationarypoint and from a height of a point of the target from the ground. Cameraroll may be computed based on respective image coordinates of at leasttwo points of the target.

According to an embodiment of the present invention there is provided, atarget usable for calibration of a camera in a vehicle. The camera isconfigured for use in a driver assistance system. The calibrationincludes setting of a camera orientation parameter and/or a cameraheight parameter. The target includes a symmetrically repeating patternof light and dark portions so that when imaged by the camera, thepattern is recognizable in an image of the camera as a series of saddlepoints. The target optionally includes markings adjacent to the lightand dark portions adapted for uniquely identifying the saddle points inthe series. The markings preferably uniquely identify the saddle pointsunder rotation of the target in the plane of the target by 180 degrees.

According to an embodiment of the present invention there is provided acamera mount for mounting a camera inside a windshield of a vehicle. Thecamera includes a lens mount and a camera housing. The front tip of thelens mount is constrained to be in close proximity to or constrained tocontact the inside of the windshield for different rake angles of thewindshield. A mechanism provides a tilt rotation axis at a point betweenthe lens mount and the windshield. The tilt rotation axis issubstantially a horizontal axis. A fixed element is mounted on theinside of the windshield. The mechanism includes a pin through the tiltrotation axis. The pin is adapted to attach to the fixed element and toconstrain the front tip of the lens mount to be in close proximity to orto contact the inside of the windshield for different rake angles of thewindshield. An arm may be adapted for attaching the fixed element to thecamera housing.

Alternatively, an element may be adapted for mounting on the inside ofthe windshield. The element has a semi-circular cross-section in avertical plane. The tilt rotation axis is substantially at the center ofthe semi-circular cross section. The camera housing is disposed betweenthe windshield and the element of semi-circular cross section. Thecamera housing is adapted to contact the element of semi-circular crosssection at a contact point. A surface of the camera housing may bebeveled at the contact point to match a contour of the element of thesemi-circular cross section. The element having a semi-circularcross-section in a vertical plane is a portion of sphere or portion ofcylinder. Optionally, a fastener is adapted for attaching through a slotin the element of semi-circular cross section. The fastener is adaptedfor constraining the front tip of the lens mount to be in closeproximity to or to contact the inside of the windshield for differentrake angles of the windshield.

According to the present invention there is provided a method formounting a camera inside a windshield of a vehicle. The camera includesa lens mount and a camera housing, A tilt rotation axis is provided at apoint between the lens mount and the windshield. The front tip of thelens mount is constrained to be in close proximity to or to contact theinside of the windshield for different rake angles of the windshield. Afixed element may be mounted on the inside of the windshield. A pin maybe attached to the fixed element for constraining the front tip of thelens or lens mount to be in close proximity to or to contact the insideof the windshield for the different rake angles of the windshield andthe fixed element may be attached to the camera housing. An elementhaving a semi-circular cross-section in a vertical plane may be mountedon the inside of the windshield. The tilt rotation axis is substantiallyat the center of the semi-circular cross section.

The foregoing and/or other aspects will become apparent from thefollowing detailed description when considered in conjunction with theaccompanying drawing figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIG. 1a illustrates a vehicle in which a camera is mounted in accordancewith embodiments of the present invention.

FIG. 1b illustrates the camera of FIG. 1a attached to a processor toprovide a hardware platform suitable for bundling driving assistanceapplications according to embodiments of the present invention.

FIG. 2a illustrates a method for detection of and recognition of trafficsigns, according to an embodiment of the present invention.

FIG. 2b illustrates an image sensor, according to a feature of thepresent invention.

FIG. 2c illustrates, a method of setting camera parameters in a secondpartition of image frames based on a gain control in a first partitionof the image frames, in accordance with embodiments of the presentinvention.

FIG. 2d , illustrating a method of traffic sign recognition anddetection according to an embodiment of the present invention.

FIG. 3a illustrates a graph of responsivity of cyan, magenta and yellowportions of a conventional CCD image sensor.

FIG. 3b illustrates a graph of responsivity of the modified cyan portionof CCD image sensor, in accordance with a feature of the presentinvention.

FIGS. 3c and 3d illustrate schematically two examples respectively of aCYM filter modified according to features of the present invention.

FIG. 4a shows a forward view from the camera mounted inside host vehicleillustrating headway computation according to an embodiment of thepresent invention.

FIG. 4b shows another forward view from the camera mounted inside hostvehicle illustrating another embodiment of headway computation,according to an embodiment of the present invention.

FIG. 4c illustrates a method for determining headway, according to anembodiment of the present invention without having or requiring speed ofhost vehicle.

FIG. 4d illustrates another method for determining headway, according toan embodiment of the present invention without having or requiring speedof host vehicle.

FIG. 5a which illustrates schematically a rear view camera systemmounted on host vehicle, according to an aspect of the presentinvention.

FIG. 5b shows a flow chart of a simplified method for detecting fogusing a camera system, according to an embodiment of the presentinvention.

FIG. 5c illustrates a method of fog detection, according to anotherembodiment of the present invention.

FIG. 6a is a flow chart illustrating a method, according to anembodiment of the present invention for performing calibration of acamera mounted on a host vehicle for use in a driver assistance system.

FIG. 6b illustrates an example of a calibration target, according to anembodiment of the present invention.

FIG. 6c shows a pattern which may be used for camera heights rangingfrom 120 cm to about 200 cm.

FIGS. 6d, 6e and 6f illustrate details of calibration targets, accordingdifferent features of the present invention.

FIG. 7 illustrates schematically a camera and two windshields withdifferent rake angles.

FIG. 7a shows a conventional design of an adjustable camera mount.

FIGS. 7b and 7c show an embodiment of a camera mount according to thepresent invention.

FIG. 7d show the embodiment of the camera mount of FIG. 7c with amodification

FIG. 7e illustrates a second embodiment of a camera mount, according toan embodiment of the present invention.

FIGS. 7f and 7g illustrates a side view and rear view respectively, of acamera mount according to the embodiment of the present invention ofFIG. 7 e.

FIG. 7h is a flow chart illustrating a method for mounting the camerabehind the windshield according to an embodiment of the presentinvention; and

FIG. 7i , illustrates a camera mount according to another embodiment ofthe present invention.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments of the presentinvention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The embodiments are described below to explain the presentinvention by referring to the figures.

Before explaining embodiments of the invention in detail, it is to beunderstood that the invention is not limited in its application to thedetails of design and the arrangement of the components set forth in thefollowing description or illustrated in the drawings. The invention iscapable of other embodiments or of being practiced or carried out invarious ways. Also, it is to be understood that the phraseology andterminology employed herein is for the purpose of description and shouldnot be regarded as limiting.

By way of introduction, the present disclosure describes “bundling” ofmultiple driver assistance systems (e.g. automatic high-beam control(AHC) and traffic sign recognition (TSR), lane departure warning (LDW),forward collision warning (FCW)) on a single hardware platform, e.g.camera and processor. Bundling provides cost reduction and may allowmore driver assistance functions to be added to the vehicle withoutincreasing the space required beyond the windshield of the vehicle.Different driver assistance applications have different requirementsfrom the camera, i.e. image sensor and optics. For example, aconventional automatic high-beam control (AHC) algorithm makessignificant use of color information and thus requires a color sensor,while lane detection and traffic sign recognition require the extrasensitivity of a monochrome sensor for operation under low lightconditions. No single gain/exposure setting is optimal for allapplications and in fact, some applications (such as AHC and TSR) mayeach use more than one gain/exposure setting, according to differentfeatures of the present invention.

Referring now to the drawings, reference is now made to FIGS. 1a and 1bwhich illustrate a driver assistance system according to embodiments ofthe present invention. FIG. 1a illustrates a vehicle 108 in which acamera 104 is mounted typically just inside the windshield near thecenterline of vehicle 108 and close to the rear-view mirror attachment.A camera mount 70, according to embodiments of the present inventionfacilitates mounting camera 70 in close proximity to the windshield.Referring now also to FIG. 1b , camera 104 is attached to a processor100 to provide a hardware platform 10 suitable for bundling drivingassistance applications according to embodiments of the presentinvention. Camera 104 includes optics, e.g. lens 105 and an image sensor106. A patterned, i.e. spatial color filter 30, according to embodimentsof the present invention is located in the focal plane of lens 105.Image frames 102 are shown schematically which are transferred fromimage sensor 106 to processor 100 for simultaneously performingdifferent driver assistance functions, according to embodiments of thepresent invention. Processor 100 is configured to operate according to anumber of algorithms. After mounting, camera 104, a calibrationprocedure 62 is used to calibrate angular orientation, i.e. pitch, yawand roll of camera 104, and to find camera height. A target 60 used forthe calibration (process 62) of camera 104 is shown in FIG. 1a ,according to embodiments of the present invention. Other processes forwhich processor 100 is configured, according to embodiments of thepresent invention are: traffic sign detection 20; electronic trafficsign recognition 22, fog detection 50, and a method 40 for determiningheadway 40.

According to an exemplary embodiment of the present invention. Imageframes are partitioned into three slots or partitions so that threeimage frames (frame 1, frame 2 and frame 3) are captured in sequencewith different image capture parameters (e.g gain, exposure time). Acomplete set of three frames is captured typically in 66 milliseconds.Image frame 1 which has a relatively long exposure is primarily used forlane departure warning (LDW) and vehicle detection. Clear lane marks aredetected even in low light conditions. However, the relatively longexposure produces motion blur on traffic signs which makes traffic signsunreadable or unrecognizable. Furthermore, distant oncoming lights tendto have blurred image spots. The outline of a vehicle ahead and thevehicle point of contact with the ground may also be determined usingimage frame 1, which may then be used to compute distance andtime-to-contact according to U.S. Pat. No. 7,113,867 for forwardcollision warning (FCW). The relatively long exposure of image frame 1may also be used to detect distant taillights for automatic high beamcontrol (AHC) as disclosed in U.S. Pat. No. 7,566,851 and to detecttraffic signs as candidate circles for eventual traffic sign recognition(TSR) as disclosed in US patent application publication US20080137908.

Image frame 2 has a medium exposure and high gain. Image frame 2 hasminimal motion blur and may be used for traffic sign recognition (TSR).The gain and exposure settings used for the traffic sign recognition(TSR) image frame 2 may be previously determined or determineddynamically (on-the-fly). Once the outline circle of a traffic sign hasbeen detected and tracked in two images (in any combination of longerexposure image frames 1 and medium exposure image frames 2) it ispossible to predict the location and size of the sign in the next image.It is also possible to determine the maximum allowable exposure thatwill keep the motion blur below a certain limit. The exposure of imageframe 2 may be set on-the-fly to keep motion blur is to less than onepixel. In some cases the maximum allowable exposure combined with themaximum gain, give an image which is too bright and the traffic sign issaturated. In order to avoid saturation, brightness of the sign ispredicted based on the distance (derived from the image size of thecircle), the angle based on the predicted location of the sign in theimage and/or the high/low beam status. The gain and exposure are thenset accordingly. Further adjustment can be made using closed loopcontrol. Since the image frame 2 uses a high gain, it has significantimage noise and not optimal for LDW under low light and/or poorlycontrasted lane markings. However image frame 2 does give usefulinformation on well-marked highways and can effectively double the framerate on such roads which are more typical for fast moving vehicles.

Image frame 3 has a short exposure and may be used together with longerexposure image frame 1 for accurate interpretation of light sources. Forexample, oncoming headlights of one or more vehicles in image frame 1appear as a bright cluster of lights. Short exposure image frame 3 ofthe bright cluster can clearly distinguish the cluster as the oncomingheadlights.

Detection and Recognition of Non-Electronic Traffic Signs

Reference is now made to FIG. 2a which illustrates a method fordetection of and subsequent recognition of non-electronic traffic signs,according to an embodiment of the present invention. Image frames fromcamera 104 are partitioned (step 200) into multiple portions orpartitions. Long exposure times are set (step 202) for the firstpartition and shorter exposure times are set (step 206) for the secondpartition. Image frames 102 are captured, typically alternating betweenimage frames 102 of the first partition and of the second partition.Image frames 102 are transferred (step 212) to processor 100. Detectionof traffic signs is performed (step 213) using the first partition withthe longer exposure times and recognition (step 214) of the traffic signis performed using the second partition of image frames with shorterexposure. Optionally, in step 202 setting exposure times or gains forfirst partition of image frames 102 is achieved using a gain control,and exposure times (or gains) are set in step 206 for the secondpartition based on the controlled parameter of the first partition.

Detection and Recognition of Electronic and Non-Electronic Traffic Signs

The light emitting diodes (LEDs) of pulsed electronic traffic signs arenot on all the time but are pulsed on and off at a frequency oftypically 80-160 cycles/sec Hertz (Hz.). The frame exposure time may beselected so as not to oversaturate the image of the electronic trafficsign but still capture the full cycles of the switching light emittingdiodes (LED). LED frequency, duty cycle and intensity vary from sign tosign. The operating parameters of pulsed electronic traffic signs mayvary from country to country and even may vary according to the time ofday and ambient light levels. However the cycle time of most or all LEDsigns is longer than about 6 milliseconds and less than about 11.4milliseconds. Reference is now made to FIG. 2b which illustrates imagesensor 106, according to a feature of the present invention. Camera 104is previously aligned in vehicle 108 so that image regions (A, B and C)of image sensor 106 are imaged in image frames 102. Region C includes animage of the road region (immediately in front of vehicle 108), regionsA and B image the sides of the road region where electronic signs arepossibly located. Image sensor 106 has two main parameters that affectthe image brightness: exposure (also known as integration time) andgain.

Reference is now made to FIG. 2c which illustrates a feature of thepresent invention, a method of setting camera parameters in a secondpartition of image frames, e.g. image frames 2 based on a firstpartition, image frames 1 for example using the AGC of image frames 1.

To accommodate recognition for the various electronic signs and theoperating environments of the electronic signs, the exposure/gaincontrol of camera 104 has different algorithms (executed by processor100) and settings for day, dusk and night operation. For instance, indecision box 220 the decision of processor 100 between day, dusk andnight modes of image frames 2 is optionally based on the exposure/gainparameters of another image frame, typically image frame 1 ortime-to-lane crossing (TLC) frame used in LDW. A classicautomatic-gain-control (AGC) algorithm may be used for image frame 1 sothat a certain percentage of pixels in a specified image region (e.g. A,B and/or C) are within a certain value range. Typically for time-to-lanecrossing (TLC) image frame 1, the gain and exposure is set so that 93%of the pixels in the regions A, B and C are below a value of 144 (out of155).

Image frame 1 automatic gain control (AGC) is designed to keep the gainas low as possible and exposure is increased first to a maximum of about40 milliseconds. When the exposure of image frame 1 based on theautomatic gain control is above 6 milliseconds processor 100 specifies‘dusk’ mode for image frame 2 and when the exposure passes 30milliseconds processor 100 specifies ‘night’ mode. There is typically nohysteresis specified in the exposure thresholds but the exposure ofimage frame 1 preferably specifies a new setting for at least 50 framesbefore switching from day mode to dusk mode or dusk mode to night mode.

Day Mode

During the day mode, the automatic high beam control (AHC) algorithm isnot active and image frame 3 allocated (at night) for the automatic highbeam control (AHC) algorithm may be used by the traffic sign detectionand recognition algorithm in the day. During the day, the traffic signdetection and recognition algorithm typically controls the exposure andgain settings of image frame 3. During the day, image frame 1 may alsobe used by traffic sign detection and recognition algorithm but theexposure and gain are optimized for lane departure warning.

Reference is now made to FIG. 2d , a method illustrating traffic signrecognition and detection according to an embodiment of the presentinvention.

The gain control (GC) of image frames 3 during the day has two modes ofoperation shown in decision box 222. In the first mode, for traffic signdetection (step 221) when the system is looking for candidates(typically circles of the appropriate size and location) of trafficsigns an open loop gain control is used. The second mode called closedloop AGC is used to optimize the exposure for verifying and recognizing(steps 224, 226, 228) the particular sign. Even though the objects weare looking for (traffic signs) are above the horizon we perform gaincontrol in the first mode based on road region C because the area of theimage above the horizon varies significantly in brightness due to thebackground (e.g. mountains, buildings or sky). The brightness of theroad surface (region C) is generally a much better indicator of thegeneral illumination. This was a non-obvious discovery: that the AGC forTSR is best performed based on the road and not on the regions wheretraffic signs are likely to be found. The gain control algorithm fortraffic sign detection (step 221) candidate search is similar to the AGC(step 220) used for the LDW frame (image frame 1) but with differentparameters even though they both use road region C. Exposure settingsare selected so that 93% of the pixels are below 128 (out of 256). Theabove setting works well in most cases. However if the road surfaceappears washed out due to low sun in front, the algorithm parameters areadjusted and the exposure is set so that 30% of the pixels are below avalue of 111. The already bright road is nearly totally saturated.However, this may be an optimal setting for detecting traffic signs inthis situation, since the traffic signs, with the illumination frombehind, are self-shadowed and appear very dark. When the sun is low butbehind the car, the road also often appears very bright and washed outhowever in this case the traffic signs are illuminated from the frontand the illumination is not increased further. To detect the particularsituation of bright roads due to low sun, histograms of the two smallregions ‘A’ and ‘B’ are compared to the histogram of the road region C.If the low sun is behind, then the image above the horizon is bright. Ifthe low sun is in front, then the image above the horizon is dark. Theactual comparison is done by running an SVM classifier on the 32 binhistogram values of the 3 regions together with the exposure and gainsettings of the current image, the average brightness of the pixelsabove the regions A, B and C and the average brightness of the pixelsbelow the region C.

Up to here the AGC used is for both electronic and non-electronictraffic signs. After detection different strategies are used for theelectronic and non-electronic traffic signs. Once a circle candidate isdetected (decision box 222) the system is switched to a closed loop AGC.There are two types of closed loop AGC for daytime: one is for regularnon-electronic sign recognition (step 228) and one for electronictraffic sign recognition 226. So we first check in decision box 224 ifthe detected candidate is a non-electronic sign or electronic sign andchoose the correct closed loop AGC algorithm. Electronic signs aretypically bright. To test for an electronic sign in the daytime, wefirst require that a certain percent of pixels are above a certainthreshold (e.g. in saturation). For example we require that 25% of thepixels inside the candidate circle are in saturation. If the candidatepasses this first test we run an SVM classifier trained on candidatesthat were electronic signs and candidates that were regular signs as anegative training set or ‘falses’.

Closed Loop AGC for Daytime:

Non-Electronic Traffic Sign:

The gain and exposure to bring the median value of the pixels inside thesquare bounded by the candidate circle to be 150.

Electronic Traffic Sign:

The electronic sign typically flickers although the duty cycle istypically quite high in daylight. All three image frames (image frame 1(LDW), image frame 2 (TSR) and image frame 3 (AHC) are used withdifferent exposures. The TSR frame gain and exposure is set as for dayclosed loop AGC. Image frame 1 (LDW) frame uses the settings optimizedfor LDW. Image frame 3 is set to an exposure of 1.5 milliseconds unlesseither the LDW or TSR frames are set to exposure of anything between 1.2milliseconds and 1.8 milliseconds. In that case, the image frame 3 (AHC)is set to 3 milliseconds.

Night Mode:

In ‘Night Mode’ the automatic high beam control (AHC) algorithm isactive and the gain and exposure settings for the automatic high beamcontrol (AHC) image frame 3 are set by the automatic high beam control(AHC) algorithm. These are optionally a fixed cycle of four exposuresettings {6 ms 0.75 ms 3.9 ms 0.75 ms}. The gain is set to 1. Theautomatic high beam control (AHC) image frame 3 may also be used by thetraffic sign recognition (TSR) algorithm. The traffic sign recognition(TSR) frame, image frame 2 is running at a fixed exposure of 13.5milliseconds and gain of 4. Once we detect a traffic sign (typically asaturated bright circle) we first determine if it is likely to be anelectronic sign or a regular non-electronic sign and apply theappropriate automatic gain control (AGC). We should note that at nightthere are many bright circular images: not just electronic signs andregular signs but also light sources such as headlights and taillightsand even some street lights. Once we have detected a round saturatedcircle in the traffic sign recognition (TSR) frame (image frame 2) welook in the automatic high beam control (AHC) frame (image frame 3)which has shorter exposure. In the shorter exposure we would start tosee the texture inside the electronic sign which is distinct from aregular sign or a light source. The respective locations in image spacein current traffic sign recognition (TSR) frame 2 and previous trafficsign recognition (TSR) frame 2 may be used to interpolate and get aposition in automatic high beam control (AHC) frame 3. A classifier(SVM) may be used based on the raw intensity image to differentiatebetween a light source, a non-electronic traffic sign and an electronictraffic sign. If the candidate is classified as a likely non-electronictraffic sign or electronic sign the appropriate closed loop automaticgain control (AGC) is used. If more than one likely sign is detectedoptimization may be selected for the largest candidate detected, sincethis would indicate that it is likely to be the closer traffic sign. Ifno candidate was classified as a likely sign traffic sign recognition(TSR) exposure parameters are maintained at the fixed value of 13.5milliseconds with gain equal to 4.

Closed Loop Automatic Gain Control (AGC) for Nighttime:

1. Non-Electronic Traffic Sign at Night:

The traffic sign recognition (TSR) frame settings are now optimized forrecognition of non-electronic signs. The gain and exposure settings usedfor the traffic sign recognition (TSR) frame (image frame 2) aredetermined dynamically. Once the outline circle of a traffic sign hasbeen detected and tracked in two images (any combination of longexposure frames and traffic sign recognition (TSR) frames) it ispossible to predict the location and size of the sign in the next imageframe. It is also possible to determine the maximum allowable exposurethat will keep the motion blur below a certain limit. Motion blur istypically kept to less than one pixel.

In some cases the maximum allowable exposure combined with the maximumgain, give an image which is too bright and the traffic sign issaturated. In order to avoid this situation, the brightness of the signis predicted based on the distance (derived from the image size of thecircle), the angle based on the predicted location of sign in the imageand the high/low beam status. The gain and exposure are then setaccordingly. This can typically be done for OEM serial production wherethe type of headlights and windshield glass is known.

Further adjustment can be made using closed loop control. If we have alikely non-electronic sign which is overly saturated, gain and/orexposure are adjusted to a lower setting. For example, for exposure time9 milliseconds and gain 3 gives half of the brightness of the standardexposure of 13.5 milliseconds and gain 4.

At night, the automatic high beam control (AHC) frame is typically alsoused for non-electronic traffic sign recognition and a traffic signrecognition classifier is run on all image frames 1, 2 and 3.

2. Electronic Traffic Sign at Night:

Electronic signs are quite bright and the brightness of their images isnot affected by the cone of illumination of the vehicle headlights.According to features of the present invention the full LED cycle iscaptured and an image is required which is not over saturated. LEDfrequency, duty cycle and intensity vary from sign to sign. The cycletime of all LEDs is smaller than 11.4 milliseconds and longer than 6milliseconds. However an exposure of 11.4 milliseconds, even withminimal gain, sometimes leads to saturation. If the exposure time forall image frames is uniformly decreased to 4.5 milliseconds or lessthere may be frames where the sign is not captured.

According to an aspect of the present invention, one or more of theimage frame partitions is sub-partitioned into multiple exposuresettings. One or more of the frames is toggled between differentsettings. In a frame triplet when the automatic high beam control (AHC)image frame 3 uses a short exposure setting, a longer exposure is usedfor traffic sign recognition (TSR) image frame 2 and vice versa. Forexample: image frame 3 (AHC) is cycled between four exposure times of {6ms 0.75 ms 3.9 ms 0.75} and image frame 2 traffic sign recognition (TSR)is cycled. {1.5 ms 9 ms 0.6 ms 6 ms} The frame 2 exposures are thus setto compliment the frame exposures to ensure a short exposure and amedium exposure in each frame triplet.

Some back illuminated signs (such as in tunnels) are not LEDs andrequire very short exposure (0.75 milliseconds). If the AGC is in thenight Electronic sign mode there is an appropriate exposure frame everytriplet. However, back illuminated signs are often not classified aselectronic signs and the AGC is in night non-electronic sign mode. Inthis case, it is the short automatic high beam control (AHC) imageframes 3 that give the correct classification.

Dusk Mode

The typical dusk scene has the following characteristics:

-   -   Scene is sort of lit (day like). The AHC function is not active        and the AHC frame slot image frame 3 may be controlled by the        traffic sign recognition algorithm.    -   Cars are switching on lights    -   LED duty cycle changes.

If we are in ‘Dusk Mode’, as determined by the LDW frame exposure(decision box 220 in FIG. 2c ) the TSR frame (image frame 2) settingsare adjusted using the AGC day algorithm. This is a typical scenariowhere we will have to deal with low light from in front. We set the AHCframe (image frame 3) to exposure 6 milliseconds and gain 1. Since theduty cycle of electronic signs during dusk is still quite long theexposure of 6 milliseconds is good at catching the electronic signswithout saturating. Both the TSR (image frame 2) and AHC (image frame 3)frames are used for classification.

Optical System

Camera 104

Reference is now made again to FIG. 1b which illustrates a camera system10 including camera 104 connected to and passing image frames 102 toprocessor 100. Camera 104 includes lens 105, image sensor 106 and afocal plane filter 30. Camera 104 orientation may be centered around thehorizon as a compromise since the for traffic sign recognition a smallupward tilt may be preferred for better detection of overhead signs,however an upward tilt is unacceptable for lane departure warning. Thecamera height is typically about 1.2 meters above the ground.

Sensor 106

A sensor appropriate for embodiments of the present invention is a wideVGA sensor with resolution 752×480. Sensor 106 has a 6 μm×6 μm squarepixel. It has a global shutter and can switch exposure and gain settingsevery frame. One unusual aspect of camera is that it includes focalplane filter 30 with only red and clear pixels. The red pixels arearranged as they would be on a standard Bayer pattern but the green andblue filters are missing. This is done in order to combine most of theadvantages of the monochrome sensor in terms of low light sensitivityand sensor 106 resolution with some color information; in particular,the ability to distinguish between red and white lights.

Lens 105

The lens example discussed is a 5.7 mm lens with a low F number F1.6 5.7mm (e.g. Marshall Electronics Inc., El Segundo, Calif.). An infra-red(IR) cutoff lens filter may be used to give a consistent response with avariety of windshields whose transmittance characteristics in the IRspectrum can vary widely. The lens filter selected is an IR cutofffilter with the 50% cutoff set at 700 nm rather than the typical 650 nm.This lens filter lets in more light and increases the low lightperformance in detection of lanes and traffic signs under halogenheadlights in particular and also increases the detection range for redtaillights.

The sensor 106/lens 105 configuration in camera 104 gives a field ofview of 0.06° per pixel or 45°×29° over whole sensor 106. However due toassembly tolerances we will only assume that 640×400 pixels areavailable giving an effective field of view of 39°×24°. Once sensor 106and lens 105 have been fixed it is often convenient to specify the lensfocal length (F) in pixels, e.g. F=950 pixels.

Cyan/Yellow/Magenta (CYM) Filter 30

While the red/clear filter focal plane filter 30 works well, there areapplications that could benefit from the full 3-dimensional color space.For example to determine traffic light colors—green, yellow and red, orto expand the range of traffic signs that can be detected; for example,to differentiate between yellow (recommended) and white (obligatory)speed signs in North America. Color information could also be used forclassifying lane marking types (white, yellow, orange and blue aretypical some examples). In fact, without color information, some lanemark color and road surface configurations make the lane marks invisibleor with very low contrast when imaged in monochrome. These can beenhanced using a particular color filter over the whole image but thechoice of filter would be region specific. The red channel of thered/clear sensor can help in enhancing the lane marks in such casehowever the lane mark color is hard to determine. To make a trulyuniversal system a color sensor is appropriate. However, a standardred/green/blue (RGB) sensor is not recommended since the green channelis much less sensitive than the monochrome channel harming both LDW andTSR performance in low light conditions. A yellow filter (from 520 nmand up into the NIR) gives a stronger response combining the response ofboth the green and red filters in the RGB sensor. Since halogen headlights are actually quite yellow in color and have only little energy inthe blue part of the spectrum, the difference between the response of amonochrome camera to that of a monochrome camera with a yellow filter issmall.

Newer xenon or high intensity discharge (HID) lamps have a stronger bluecomponent so there is noticeable drop in sensor response with the addedyellow filter. However these headlights are so much brighter in generalthan the older halogen headlights that even with the yellow filter theresponse is strong enough. In fact, with HID headlights, even an RGBsensor can be used for LDW and TSR. The new LED technology also fallsinto this second category. In order to make best use of the availablelight energy from the halogen headlights it is important to capturelight also in the NIR region. So a near infrared (NIR) cutoff filter, ifused at all, should have a cutoff at 750 nm or even 800 nm. Typicalautomotive windshield glass has a NIR cutoff characteristic however itusually only reduces near infrared transmittance to 30%. Newer and moreexpensive glass filters the NIR transmittance down to 10%, however thisglass is usually on high end vehicle where HID headlights are used andscene brightness is not an issue. The yellow filter has the addedadvantage that it enhances the contrast, in daytime, between yellow lanemarkings and the concrete road surface often used in the USA.Understanding that yellow is the best color filter to use for low lightsituations, leads to the idea of using a CYM (Cyan, Yellow and Magenta)instead of the standard RGB. Reference is now made to FIGS. 3c and 3dwhich schematically show two examples of the color pattern of CYM focalplane filter 30, according to embodiments of the present invention. Theyellow portion (Y) replaces the green in the standard Bayer pattern sothe yellow portion has the maximum resolution using half of the pixels.The remaining pixels alternate between magenta (M) and cyan (Y). Howevera standard CYM filter is preferably modified, according to an exemplaryembodiment of the present invention.

Reference is now made to FIG. 3a which illustrates a graph ofresponsivity (V/μJ/cm²) of a CCD image sensor TC246CYM-B0 (TexasInstruments Inc.) as a function of wavelength in nanometers. While thecyan portion filters out the red part of the spectrum it has noticeableNIR response above 750 nanometers. Near infra-red response of the cyanfiltered image sensor is marked with an arrow. The significant responsewith the standard cyan filter above 750 nanometers means that a red taillight gives a noticeable response on the cyan pixels and can nottypically be differentiated from white headlights. This drawback whichwe noted in standard CYM sensors is not typically noticed since colorcameras typically have an additional filter which cut out most NIR above650 nm. Thus, for a standalone AHC application a standard CYM filterwould be suitable. However when combining with other applications suchas LDW and TSR currently available CYM sensors are not preferably usedsince these applications preferably use response of the yellow channelto NIR. Therefore, according to a feature of the present invention thecyan filter as used in standard CYM filters is preferably modified sothat it blocks out or significantly reduces the NIR from 750 nanometersto at least 800 nanometers. A graph of responsivity with a modified cyanfilter, according to a feature of the present invention is shown in FIG.3b . An arrow points to the responsivity as measured through themodified cyan filter, according to a feature of the present inventionusing the same image sensor as used for the responsivity shown in FIG.3a . For an absorbing filter, the filter may be modified by adding apigment such as iron which is used to absorb NIR in standard automotiveglass. For a reflective dichroic filter a dielectric layer design withseveral non-absorbing dielectric layers are typically used. In theexample of FIG. 3a , the minimal cyan IR response is up to about 800nanometers. The wavelength range of minimal cyan IR response may beextended to 900 nanometers, 1000 nanometers or more.

Modifying the Color Matrix Values to Account for More NIR

In some applications the there is a desire to display the images to thedriver in some form. This might be the case for a backup display or anall-around-view application.

With the modification of the cyan filter, significantly more NIR reachesthe sensor the image will have a strong yellow tint if standard colormatrix values are used to convert to RGB for display. It is quite easyto manually tune the color matrix values for each scene so as to givegood color rendering however these matrix values will require retuningwhen the light source changes spectrally. In particular, settingssuitable for daylight are not suitable for a scene illuminated withhalogen lights with a spectrum skewed much more towards NIR. Accordingto an embodiment of the present invention, switching is performedbetween multiple color matrices according to light level as measured bythe camera and other information such as time and/or place typicallyavailable on the car network.

Using Monochrome Data to Train a Classifier to Work with a CYM Sensor

The Mobileye AWS 4000 system uses a monochrome sensor. The algorithmuses, among other algorithms, various classifiers which were trainedusing hundreds of thousands of training samples. Accumulating such alarge database of much time and effort. In order to avoid having such aneffort going to waste when switching to the CYM sensor the same examplesare reused. It should first be noted that the images captured from amonochrome camera or from a monochrome camera with a yellow filter arevery similar. The differences are negligible. However when using the CYMcolor filter array the response of the cyan and magenta pixels will bedifferent from the yellow pixel response creating spurious edges. Giventhat the color of the viewed object is not known the cyan and magentavalues cannot be adjusted automatically to give the correct yellowresponse. The solution is to use only the yellow pixels as input to theclassifier. The values at the cyan and magenta pixel locations areinterpolated based on the yellow neighboring pixels. These images willstill appear different from the images obtained from the monochromesensor however they can correctly be simulated from the monochrome data.As a training set for the classifiers using the CYM sensor we use themonochrome images where we interpolate in a checker board fashion. Weuse the ‘white’ pixels as is and interpolate the ‘black’ to get oneexample image and then use the ‘black’ to interpolate the ‘white’ to geta second example image.

Computing Headway

The term “headway” as used herein refers to the time it takes a hostvehicle to reach the preceding vehicles current position. The term“headway distance” as used herein refers to the distance from the frontof the host vehicle to the preceding vehicle's current position.Maintaining a safe headway distance is a key element of driving safety.Since a safe headway distance is related to vehicle speed, headway istypically in seconds or distance/speed. However computing headway basedon distance/speed requires good estimates both of distance and vehiclespeed. The distance can be estimated from an image stream obtained froma monocular camera. The vehicle speed may be obtained by connecting tothe vehicle controller area network (CAN) bus or by connecting directlyto the speedometer analog signal. However, both methods for obtainingthe speed require a professional installation. It would be advantageousfor an after-market product to avoid having to use professionalinstallation.

Reference is now made to FIG. 4a which shows a forward view 402 a fromcamera 104 mounted inside host vehicle 108 (FIG. 1a ) according to anembodiment of the present invention. Camera 104 is preferably amonocular camera mounted close to the rear mirror, looking forward outthrough the windshield of host vehicle 108. Forward view 402 a shows twovehicles 408L and 408R in two lanes and the progression of a feature 404a-404 n (behind vehicles 408) in the road surface for successivelycaptured image frames 102. Examples of features 404 a-404 n in the roadare typically the end of a lane mark segment, shadows or tar seams.

One method to determine headway is to estimate the distance of vehicle408R ahead and then divide by the speed of host vehicle 108. This willgive the time it takes host vehicle 108 to reach the current position ofvehicle 408R.

Reference is now also made to FIG. 4c showing a method 40 a fordetermining headway, according to an embodiment of the present inventionwithout having or requiring speed of host vehicle 108. Method 40 a isimplemented by an algorithm in processor 100 which determines the timefor host vehicle 108 to reach the current position of preceding vehicle408R. Method 40 a locates (step 420) a feature point 404 a on the roadsurface that is on the same horizontal image line as the bottom ofpreceding vehicle 408R. At this point in time, feature point 404 a onthe road and the preceding vehicle 408R are at the same distance fromhost vehicle 108. Feature point 404 a is then tracked (step 422) in thenext few frames (the same feature point now labeled 404 b in FIG. 4a )till the bottom of the image frame 102 is reached i.e. feature point 404n. The time stamp of the first image frame 102 showing feature point 404a is then subtracted (step 424) from the time stamp of the image frame102 where the feature point 404 n has reached the bottom of image frame102. The subtraction (step 424) between the time stamp of the firstimage frame 102 and the time stamp of the image frame 102 where thefeature point 404 n has reached the bottom of image frame 102 gives anestimate (step 426) of the headway in seconds.

It is preferable to use a scale invariant feature transform (SIFT)features for tracking (step 422) features. Since these features encodescale invariant information they are well suited for tracking featureson the road as they grow bigger. When tracking features on periodicstructures (such as dashed lane markings) a fast frame rate ispreferably maintained for frames 102 to avoid aliasing.

Referring back to FIG. 4a , camera 104 is positioned inside host vehicle108 and mounted close to the rear-view mirror, viewing in the forwarddirection through the windshield of host vehicle 108. The trackedfeature point 404 n leaves the image frame when the feature point is infact a few meters in front of host vehicle 108, a fact which addsinaccuracy to the estimated headway (step 426). Moreover, a wait of afew seconds may be needed until the headway estimate is obtained (step426); by then the distance to the preceding vehicle 408R or the speed ofthe host vehicle 108 may have changed. A wide angle camera may bemounted at the front of vehicle 108 so that in typical situations, thecorrect distance to the preceding vehicle 408R is resolved.

Reference is now made to FIG. 4b which shows another forward view 402 bfrom camera 104 mounted inside a host vehicle 108 according to anotherembodiment of the present invention. Forward view 402 b shows twovehicles 408L and 408R in two lanes and the progression of two pairs offeatures 410 and 412 (behind vehicle 408R) in the road surface forsuccessively captured image frames 102.

Typically features 410 and 412 in view 402 b are provided from onedashed lane marking and one solid lane marking respectively. If bothlane markings are dashed then the horizontal line extending from afeature detected lane mark might not intersect a lane mark segment onthe other side. In that case the intersection point of the horizontalline and a line joining the end points of two lane mark segments isused.

A substantially horizontal distance d₁ separates features 410 and 412and a substantially horizontal distance d₂ separates features 410 and412 in successively captured image frames 102. Examples of features 410and 412 in the road are typically the end of a lane mark segment,shadows or tar seams.

Reference is now also made to FIG. 4d showing a method 40 b, accordingto another embodiment of the present invention. Method 40 b firstlocates (step 430) a feature point 410 on the road surface that is onthe same horizontal image line (d₁) as the bottom of the precedingvehicle 408R. At this point in time the feature point 410 on the roadand the preceding vehicle 408R are at the same distance from hostvehicle 108. A second feature point 412 is located (step 430) along thesame horizontal image line (d₁) as the bottom of the preceding vehicle408R. Typically, linear features such as a solid lane mark are used forproviding the second feature point 412 since only the horizontallocation of the second feature point 412 is needed. Typically bothfeatures 410 and 412 are tracked (step 432) in every captured imageframe 102. From tracking (step 432), image distance (for example d₁)between the two features (features 410 and 412) can be determined. Fromthe change in image distance (i.e. d₁ compared to d₂), thetime-to-contact (T_(ttc)) can be estimated. In effect, a forwardcollision warning (FCW) is being determined not from vehicle 408R aheadbut from distance d₁, where the host vehicle 108 is in the current imageframe 102, hereinafter the first image frame.

Once the first image frame is produced both feature points 410 and 412are tracked (step 432) in the next few image frames 102 until the changein distance between successively tracked feature points 410 and 412 issignificant (for example if the size of feature points 410 and 412doubles). Image frame 102 that produces the situation where the distancebetween successively tracked feature points 410 and 412 is significantis known as a second image frame.

Once the second image frame is obtained, the time stamp of the firstimage frame is subtracted (step 434) from the time stamp of the secondimage frame. The subtraction between the time stamp of the first imageframe and the time stamp of the second image frame is known as the timedifference (δT). The time to contact (T_(ttc)) is then computed (step436) using:

$T_{ttc} = \frac{d_{1}\delta\; T}{d_{1} - d_{2}}$

The term

$\frac{d_{1} - d_{2}}{d_{1}}$is known herein as the relative change of distance in image space. Thetime-to-contact T_(ttc) or headway is then given by the time differencefound for example from the difference between the time stamps of theimage frames divided by the relative change of distance

$\frac{d_{1} - d_{2}}{d_{1}}.$

Typically time to contact (T_(ttc)) can be improved further by usingmeasurements from multiple or all the image frames 102 (i). Usingmeasurements from all the image frames 102 (i) indexed with integer i aplot of 1/d_(i) as a function of δT_(i) for each value of time (T) maybe made. The points of the plot can be fit with a linear fit producing aline and the point of intersection of the line with the time (T) axiswill give the time to contact (T_(ttc)) or headway distance. The linecan also be fit to a parabola to account for acceleration of hostvehicle 108.

In typical driver assistance systems, the headway warning is not anurgent signal. Therefore one can compute the time to contact (T_(ttc))starting at number of image frames 102 and then give a warning only if alarge number of measurements agree.

If speed for host vehicle 108 is available then the headway computedusing distance/speed may be combined with the headway computed usingmethod 40 a and/or 40 b to give a more robust warning signal. Using hostvehicle 108 speed (V) and time to contact (T_(ttc)) derived (step 426 or436) using method 40 b an estimate of the distance (Z) to the vehicle408R in front may be obtained using:Z=V×T _(ttc)

The distance (Z) does not require knowledge of the camera 104 height orthe pitch angle of camera 104. The distance (Z) can then be combinedwith the distance obtained using FCW (U.S. Pat. No. 711,386) to get amore robust distance measurement using, for example, a Kalman filter.

Fog Detection

Fog detection is an important feature when driving at night since, inthe presence of fog, a different light strategy is used. The driverswitches off high beams and switches on fog lamps if available. Thelight strategy during fog is used because light scattered from the highbeams in the presence of fog may blind the driver. Not all drivers areaware of this however, and often respond incorrectly to the reducedvisibility due to the fog by switching on their high beams. By so doing,they reduce the visibility even more. A fog detection system which canadvise the driver or switch lights on and off automatically would beuseful. There has been interest in adding a fog detection feature to acamera based driver assistance systems. One method uses spatialgradients in the image. During heavy fog the image contrast drops and sothe image loses its sharper edges. However, the presence of sharp edgesmay also be dependent on other characteristics. Urban scenes have manysharp edges while country scenes have softer edges. Furthermore, mist orcondensation on the glass may also create a drop in image contrast andreduce the higher spatial frequencies. Another method may use therelative drop in contrast of a lane mark in higher portions of theimage, i.e. as the viewing direction approaches the horizon where thelane mark is farther from the camera. In foggy conditions, a freshlypainted lane mark close to the vehicle is quite sharp but the sharpnessdrops quite quickly for more distant lane marks. It is also possible tomonitor the increase in contrast on objects, such as road signs, or lanemarks, as they get closer. If a lane departure warning system is alreadyinstalled, then such fog detection techniques could be used. Thesemethods assume lane markings of good quality or an object in the scenethat can be tracked, and they may require significant computationalpower.

A more direct measure of the driver's need to lower high beams is theamount of scatter produced by the atmospheric conditions. If the sceneis dark then it is possible to detect the light of the host vehiclehigh-beams scattered by the fog back to the camera. The high beams havea very particular back scatter pattern which is determined by thelocation of the forward looking camera and the vehicle headlights. Theback scatter pattern from the high beams of the host vehicle may belearned by using machine learning techniques. The input feature vectormay be a few horizontal lines from a low resolution image. The problemis that low beams do not give rise to strong scatter pattern. This isdue to the fact that the low beams illuminate the close road which alsoreflects back to the camera. There is not a significant difference inthe reflection pattern from the road or the scatter pattern from the lowbeams. This means that it is hard to give the driver, who has switchedto low beams due to the fog, an indication that the fog has ended andnow is a good time to switch back to high beam.

This method may be less applicable in an illuminated night scene.However one might argue that in an illuminated night scene an averagedriver would use low beams. Fog is readily detectable when other lightsources such as street lights are in the scene or other cars with theirlights on (high or low) are viewed from the side.

Fog Detection Using a Rear Camera

Some car models are available with rear-view camera systems. Referenceis now made to FIG. 5a which illustrates schematically a rear viewcamera system 50 mounted on host vehicle 108, according to an aspect ofthe present invention Rear camera 104R is shown mounted on the rear ofvehicle 108, typically centered along the rear of vehicle 108 andoriented backward. One use of rear camera system 50 is for detection ofobstructions behind vehicle 108. Obstruction detection using rear camerasystem 50 is disclosed in co-pending US patent application publication20080266396 and international application PCT/US/08062003. Thehorizontal field of view (HFOV) of camera 104R is large between 130 and180 degrees. Rear-view camera may view almost the entire cone of lightemanating from the host vehicle taillights 107. The cone of light fromtaillights 107 is shown schematically as between two dotted lines. Thus,rear camera 104R is well located to observe light scatter fromtaillights 107 due to fog 51 shown as a droplet of water. An incidentlight ray i is shown and a scattered light ray s scattered into theaperture of camera 104R. In the absence of scattered light from fog 51,at least a portion of the image of image sensor 104R is not illuminatedby taillights or background light at night. Furthermore, the lightemitted from the taillights is red. Thus the detected scattered lightwill have a red hue which differentiates the detected scattered lightfrom white background light. Since there are two taillights 107 andcamera 104R is symmetrically situated between two taillights 107, asymmetric pattern of detected light is expected due to scattering fromfog 51. We will further note that in original equipment manufacture(OEM) series production the location of camera 104R and taillights 107is known thus the scattered light in the presence of fog 51 causes anexpected shape in the image plane of image sensor 104R.

Detection Using Rules

It is possible to engineer a rule based system that looks for particularregions in the image being brighter than other regions and that the redhue is above a certain threshold. One then determines if there is asignal on both left and right edges of the image then fog 51 isdetected.

Detection Using Machine Learning

Alternatively machine learning techniques are used to determine thepresence of fog 51. Reference is now made to FIG. 5b which illustratesschematically by way of flow chart a simplified method 52 for detectingfog 51 using camera 104R or camera 104, according to an embodiment ofthe present invention. A large number of image frames 501 are used fortraining (step 503) in order to produce a trained classifier 505.Trained classifier 505 is subsequently used to classify (step 509) adetected halo as fog when the halo is detected (step 507)

The image is reduced to 160×120 red/green/blue RGB pixels (i.e. the red,green and blue pixels are averaged separately to preserve colorinformation) and optionally only the left and right 30 columns of thetop half of the image are used for fog detection. The result is afeature vector of 80*(2*30)×3 values.

The machine learning technique as used was the support vector machine orSVM. A linear kernel can be used and trained on 100,000 frames 501 ofwhich half are examples of fog taken from multiple vehicles 108 from thesame car model. Other machine learning techniques such as principalcomponent Analysis and Fisher discriminants also show promising results.

Since we do not require very fast response information of many framesmay be accumulated such as 90 frames (3 seconds at 30 frames per second)and since flickering or oscillating states of headlights and/or foglights is undesirable a constraint may be imposed that a fraction, e.g.˜80% of the frames agree in order to switch state of fog lights and/orheadlights.

Modification of Front Lights and Using Forward Camera 104

Reference is now made again to FIG. 5a which illustrates anotherembodiment of the present invention. Headlight 109 of vehicle 108 isillustrated. A modification of headlight 109 causes a small amount ofthe light to be diverted upward. A single light ray i is shownpropagating upward from headlight 109. In that case the forward lookingcamera 104 will simply looks for a vertical beam of light. The verticalbeam is visible only when there is fog 51 or another scattering mediumin the atmosphere. A single ray s scattered from fog 51 is shown.

Other Configurations

Using Change in Light Status

It would advantageous to have additional information about any change inrear light status such as braking, reverse or even turn signals. Anychange in brightness that corresponds in time to changes in rear lightstatus may give further indication as to scattering.

Reference is now made to FIG. 5c which illustrates a method 53,according to another embodiment of the present invention. Certain LEDtaillights flicker at a high frequency. By synchronizing (process 54)video capture to the switching frequency of taillights 107 one cansequentially capture images (steps 523) with taillight illumination andcapture images (step 527) without taillight illumination. It would alsobe possible for the system to initiate a brief flickering of thetaillights even in the case of regular incandescent taillights toachieve a similar effect. Taillights or other illumination may beswitched on (step 521) and very briefly switched off (step 525).Subtracting (step 529)) or otherwise comparing or correlating the two ormore captured images) may include a subtracted image of the lightscattered with an enhanced the signal to background ratio for use in fogdetection (step 531) based on light scatter in the subtracted image.

Using a Dedicated Light Source

According to another embodiment of the present invention, a speciallight source such as a LED with a focused beam may be mounted on the carhood in front of the camera with the beam facing up. Since it is adedicated light source it can be alternatively switched on and offsynchronously with image frames of either camera 104 or 104R and thenimage subtracted the light scattered from the dedicated light source maybe sensed.

Single Pole Calibration

Reference is now made to FIG. 6 which includes a flow chart illustratinga method 62, according to an embodiment of the present invention forperforming calibration of camera 104 mounted in vehicle 108 for use in adriver assistance system. Method 62 uses upright target 60 (FIG. 1a )placed at two distances from vehicle 108. Target 60 is first placed(step 601) at a first distance, e.g. up against the front bumper ofvehicle 108 on the centerline of the vehicle. Standard features on thevehicle may be used such as the license plate or the car makers emblem.Typically the first distance is zero centimeters—the target is placedtouching the front bumper centered on the bumper.

A first image is captured (step 603) of target 60. Target 60 isrelocated (step 605) to a second distance, e.g. moved back away fromvehicle 108 approximately one meter. The lateral location of the targetin the second position is not important. A second image is captured(step 607) of target 60. A mapping (step 609) is performed betweenrespective image coordinates of the first image and the second image. Astationary point during mapping (step 609) is a focus of expansion (FOE)typically on the horizon. This provides the Y coordinate of the FOE usedin the algorithms. The lateral position of the target in the first imageprovides the X coordinate of the FOE used in the algorithms. If thecamera is not located in the center of the image this is adjusted for.and adjusted to correspond to the center of vehicle 108. Method 62 mayprovide the camera orientation in all three angles and the cameraheight.

Example of a Makeshift Target Assembly for Initial Tests

Reference is now made to FIG. 6b which illustrates an example of acalibration target 60, according to an embodiment of the presentinvention. Target 60 includes an optical pattern 615 b mounted forinstance on a rigid upright pole 610. The base 613 is preferablysufficiently heavy so that it does not wobble. It is recommended thatbase 613 have three points of support to insure stability. Pattern 615optionally includes a narrow checker pattern, an example of which isshown in FIG. 6b and enlarged in FIG. 6d . In this case the squares areoptionally five centimeters on the edge. Another example of pattern 615is shown in FIG. 6e . The narrow shape is tends to keep target 60 morestable and to reduce the chance of being blown over by wind. The heightof target 60 is designed to be wholly visible when placed close to thebumper of vehicle 108. A different target height is typically requiredfor trucks and automobiles.

Target Size and Height

Reference is now made again to FIG. 6d which illustrates pattern 615 bwith six rows of alternating black and white squares. Five saddle points617 are shown marked with small circles with dotted lines while only thetopmost saddle point is labeled with the reference number 617 so as notto clutter the drawing. With an additional five centimeters of widemargin above and below the total target is 40 centimeters high. However,only the middle 30 cm may be visible in the image of camera 104.

Given that target 60 is located about 1.5 meters from camera 104 and thetarget height is 30 centimeters, the image height is f·(30/150)=95pixels with a camera lens of focal length 475 pixels. The distance inimage space between top and bottom saddle points is 79 pixels. Giventhat the total image height is 240 pixels we can afford to have thecamera height vary about 30 cm without requiring adjustment of thecamera height. Thus we may have one target for camera 120 cm-150 cm anda second for camera height 150 cm-180 cm.

For camera height 120 cm to 150 centimeter the center of target 60should be at 120 centimeter. The bottom at 105 centimeters (bottomsaddle point is at 110 centimeters). This value is based on thefollowing reasoning: If target 60 is placed 1.5 meters from camera 104and target 60 is 30 centimeters in height then about 30 centimetersextra is available to adjust camera height. Assuming target 60 works forvehicles of height within 120 cm to 150 cm then the center of target 60is at 135 centimeter.

L=30 centimeter target length (six rows of five centimeters in FIG. 6d )

dH=30 centimeter variation in camera height (e.g. from 120 cm to 150 cm)

F=475 focal length in pixels

Z=150 centimeter target longitudinal distance from camera 104

V=vertical extent of target in the image for the camera height variationdH

$V = {{F\frac{\left( {L + {dH}} \right)}{Z}} = {\frac{475 \times \left( {30 + 30} \right)}{150} = {190\mspace{14mu}{pixels}}}}$

The image of pattern 615 b of FIG. 6d is 240 pixels high so a margin oferror and tolerance is available of about 50 pixels (i.e. 20 to 70pixels).

The distance between uppermost and lowermost saddle points is 80 pixelswhich is typically adequate for roll computation even if target 60 is1.7 m away.

The discussion above does not take into account camera angle (pitch).Camera 104 has been assumed to be horizontal. Consider that camera 104is pointed down 20-70 pixels. Taking the 5 middle pixel (i.e. 45 pixels)at 1.5 meters, 45 pixels in image space translates in real space to:

$\frac{\left( {150\mspace{14mu}{{cm}.} \times 45\mspace{14mu}{pixels}} \right)}{\left( {475\mspace{14mu}{pixels}} \right)} = {14\mspace{14mu}{{cm}.}}$

So target 60 is located 14 cm lower when camera 104 points downequivalent to 45 pixels on the average in image space. Rounding the 14cm to 15 cm, the center of target 60 is at height 120 centimeters. Thebottom of target 60 is at height 105 centimeters and the bottom saddlepoint is at height 110 cm. A second target height may optionally be usedto cover camera heights from 150 cm to 180 cm. The second target heightmay be 30 cm. higher than the first target height.

A Single Target to Cover the Whole Range

It would be advantageous to avoid having multiple target heights fordifferent camera heights, when camera 104 is mounted on eitherautomobiles and trucks. Reference is now made again to FIG. 6c whichillustrates further features of the present invention. Pattern 615 cshown in FIG. 6c may be used for camera heights ranging from 120 cm toabout 200 cm. corresponding to both smaller automobiles and trucks.

Reference is now made to FIG. 6f which illustrates pattern 615 c in moredetail. A center portion of 6 rows such as that marked with reference618 of pattern 615 c is optionally used for performing method 62. Twoadditional strips 619 allow for unique identification of saddle points617.

If target 60 is placed at a distance greater than 1 m at least 50 cmheight of pattern 615 c is imaged, corresponding to nine saddle points617. Pattern 615 c includes an innovative feature so each set of sixsaddle points has a unique code based on strips 619. With nine saddlepoints 617, at least one transition in strips 619 is imaged and verticalposition within pattern 615 c may be determined. Note that the cornersof strips 619 align with the corners of the black squares to avoidadditional saddle points which may be confused with saddle points 617.Pattern 615 c is preferably limited in height to about 1 meter so thatit does not become to long for packing. Pattern 615 c is also optimallyof length similar to the longest element of stand 610 so that pattern615 c fits in the same package for shipment. Alternatively, the lengthof pattern 615 c may be extended to 1.6 meters or 1.8 meters withoutadding another vertical stripe for coding saddle point 617 positions.

An alternative embodiment of the present invention, pattern 615 may beinverted around a rotation point 616 high up on pattern 616. Markings619 in pattern 615 c are designed to be unique under target rotation.

Code Using Strips 619

Let D be the image distance between saddle points 617. For each saddlepoint 617 we look at the regions 0.5 D to above and 1.5 D to the leftand right and check if that region is white or black:

L=1 if the color up and to the left is black

L=0 if the color up and to the left is white

R=1 if the color up and to the right is black

R=0 if the color up and to the right is white

We can now compute a code C for each saddle point 617 in pattern 615 c:C=2·L+R

For pattern 615 c the codes for the 17 saddle points 617 listed from topto bottom are: 33333222220000022. As one can see each sequence of fivecodes is unique. If pattern 615 c is inverted then the codes are:11100000111113333. Again, every sequence of five codes is unique. If wedo not know the orientation of pattern 615 c we look at six codes to geta unique sequence.

Algorithm with Example

The method first detects saddle points where two black squares meet.These are then matched between the two target images. The verticalpositions of the points in the two images are used to estimate thestationary point of the horizon, focus of expansion (FOE) Y or horizon.The lateral position of the close target is used to estimate the lateralposition of the calibration, focus of expansion (X) or Yaw.

If camera 104 is mounted in the center of vehicle 108 then the average Xposition of the point can be used as is. The lateral position of thepoints in the far pole position is ignored since target pole 50 b is notcentered on vehicle 108. If camera 104 is mounted with an offset to thecenter this is preferably accounted for. The vertical spacing of thepoints in the close image is used as a ruler since it corresponds to 5cm in the real world. Let the measured camera offset be 7 cm and thelength of the edge of the square in the close image be 10 pixels. Thenthe lateral position correction X_(offset) is calculated:X _(offset)=7/5·10=14 pixelsLocating the Target Points in Each Image

Saddle points are known to be the local minimum of the determinant ofthe Hessian. The first step is to find these local minima with a smallthreshold. Note, this measure is orientation invariant. This can becomputed on a 320 by 240 image.

The second step is to verify these points using normalized crosscorrelation with a template of a simple checker pattern with two blackand two white squares. The size of each square is kept small: typically5 pixels for the side. Only points with a high enough (positive ornegative) score are kept. The results for the close image are shown inby blue points in the bottom left image. Note one spurious point on thedark vehicle parked on the right.

The third step is to find N=5 points that are vertically aligned. Werequire that they lie within a band of 5 pixels. We further check thatthey have alternating orientation starting from the top with the uppersquare off to the left. Other tests such as requiring that the pointsare evenly spaced can also be used (but not currently implemented).

All such sets of points are detected however we expect only one such setper image. More that one set should generate an error. The right handimages show the detected points.

The determinant of the Hessian is:

${\begin{matrix}I_{xx} & I_{xy} \\I_{xy} & I_{yy}\end{matrix}}\quad$

The x and y derivatives are computed using the Sobel filter:

${dx} = {{\begin{bmatrix}{- 0.25} & 0 & 0.25 \\{- 0.5} & 0 & 0.5 \\{- 0.25} & 0 & 0.25\end{bmatrix}\mspace{14mu}{dy}} = \begin{bmatrix}0.25 & 0.5 & 0.25 \\0 & 0 & 0 \\{- 0.25} & {- 0.5} & {- 0.25}\end{bmatrix}}$

We normalize each point by the magnitude of the gradient.

$\frac{1}{1 + {\overset{\sim}{I}}_{x}^{2} + {\overset{\sim}{I}}_{y}^{2}}$

where we use Ĩ_(x) to denote the blurred versions of the imagederivative. Blurring was done by twice convolving with the filter:

$\begin{bmatrix}1 & 2 & 1 \\2 & 4 & 2 \\1 & 2 & 1\end{bmatrix}.$

We compute the local minimum of the image by computing the local maximumof the negative of the image. The function returns true for points thatare above a certain threshold t and that are greater than all the pixelsin a square region of 2·w+1 centered around the point. w is set to 4 andt is set as the standard deviation of the determinant of the Hessianimage.

Points are verified to be saddle points if their normalized 2Dcorrelation with a saddle shaped template is greater than 0.7 (orsmaller than −0.7). The correlation function is based on Matlab corr2.m.The template is a 22 checker board pattern with squares of size 6pixels. It is intentionally orientation dependent.

Detecting Vertically Aligned Points

1. For each x from 1 to 320

(a) Count the number of points that lie in a 5 pixel column around x

(b) If greater or equal to N

-   -   i. Sort points in y    -   ii. Starting from top point    -   iii. Check if N points it matches sequence of left right        template. If yes, then mark points as used and return points as        line.    -   iv. Go to next unused point.

2. If number of lines equals one then return line with ‘success’, elsereturn ‘fail’.

Note: The scan over x could go from min(x_(i)) to max(x_(i)).

Matching Points Between the Two Images

If we are using the simple case where the target is wholly visible inboth images then matching is straightforward: the points are matched oneto one starting from either end. In the case that only a subset of thepoints on the target is visible, we use the unique codes. For each imagewe compute the sub-code for the subset of points that are visible. Wethen find the location of the sub-code within the full code of thetarget. This in turn allows us to give the correct index to each pointin the subset. We use only points from the two images with matchingindices. Points in one image which do not have a match in the otherimage are not used. Alignment of the sub-code within the target alsogives the true height of each point from the ground.

Computing FOE and other Parameters from Two Lines

The lateral position of the FOE (or ‘yaw’) is given by the lateralposition of the close target. We take the average of the x coordinate ofthe points on the close line. For the horizon we are looking for thestationary point y_(o) in the mapping from the y coordinates of the linepoints in one image y₁ to the matching line points in the second imagey₂.

The model is a scale around a stationary point thus:y ₂ =S(y ₁ −y ₀)+y ₀

This can be rewritten as a linear set of equations:

${\begin{pmatrix}{y_{2}(1)} & 1 \\\vdots & \vdots \\{y_{2}(N)} & 1\end{pmatrix}\begin{pmatrix}S \\{\left( {1 - S} \right)y_{0}}\end{pmatrix}} = \begin{pmatrix}{y_{1}(1)} \\\vdots \\{y_{1}(N)}\end{pmatrix}$

We solve using the pseudo inverse:

$\begin{pmatrix}S \\{\left( {1 - S} \right)y_{0}}\end{pmatrix} = {\begin{pmatrix}{\sum\limits_{i = 1}^{N}{y_{2}(i)}^{2}} & {\sum\limits_{i = 1}^{N}{y_{2}(i)}} \\{\sum\limits_{i = 1}^{N}{y_{2}(i)}} & N\end{pmatrix}^{1}\begin{pmatrix}{\sum\limits_{i = 1}^{N}{{y_{2}(i)}{y_{1}(i)}}} \\{\sum\limits_{i = 1}^{N}{y_{2}(i)}}\end{pmatrix}}$

It is then straightforward to solve for y_(o).

Adjusting for Camera Lateral Offset

If the camera is not centered in the car it is necessary to adjust forthe camera lateral offset. We use the known size of the squares (5 cm)and the vertical spacing between the points in the close image toconvert from cm to pixels:

${F\hat{O}E_{x}} = {{FOE}_{x} + {C_{x}\frac{T_{pixels} - B_{pixels}}{T_{cm} - B_{cm}}}}$

where C_(x) in the camera lateral offset in cm and T_(pixels),B_(pixels), T_(cm) and B_(cm) are in the position of the top and bottomsaddle points in the image and in centimeters from the groundrespectively.

Computing Camera Height

To compute camera height we convert the coordinates of the stationarypoint y_(o) (also known as FOE_(y) or horizon) to height from the floor.

$H = {B_{cm} + \frac{\left( {y_{0} - B_{pixels}} \right)\left( {T_{cm} - B_{cm}} \right)}{T_{pixels} - B_{pixels}}}$Computing Camera Roll

The camera roll is computed from the close target:

${Roll} = {{arc}\;{\tan\left( \frac{X_{T} - X_{B}}{Y_{T} - Y_{B}} \right)}}$

where Xr, XB, Yr and YB are the x and y coordinates of the top andbottom saddle points respectively.

Camera Mount

In original equipment manufacture (OEM) installations there is often aspecific host vehicle model and since the rake angle of the windshieldglass is known in the mechanical design phase, camera mount 70 (FIG. 1a) can be optimized. For after market products, system 10 is potentiallyinstalled in a variety of vehicle models with a large variation inwindshield rake angles.

Reference is now made to FIG. 7 which illustrates schematically rakeangles of windshields 701 a and 701 b of different vehicle models andcamera 104. The camera elevation or pitch is preferably within a desiredtolerance angle. According to an embodiment of the present invention,camera mount adapters may be used for each possible windshield rakeangle. In practice a single adapter might cover a range of 3 degrees.For each car model, the appropriate adapter is affixed to the vehiclewindow and camera 104 is inserted into the adapter. The advantage ofthis method is that the mount can be optimized for the particular angleand bring camera 104 lens close to the windshield glass. A disadvantageis that it requires to supply the after-market unit with a variety ofadapters, most of which are not used. An alternative solution, accordingto another embodiment of the present invention is to use an adjustablemount that can accommodate a variety of angles and is adjusted afterfixing the mount to the glass so as to bring the camera to the desiredangle. While the angle of camera 104 is correct, the position of camera104 relative to the glass is not optimal and the distance can be quitelarge, especially for sharp rake angles. This in turn significantlyincreases the size of the light shield often attached around camera 104;the light shield is recommended for eliminating stray light reflectinginto the camera aperture from the windshield glass. Various mountdesigns are presented, according to different embodiments of the presentinvention which allow both adjustment of the angle and ensuring thatlens 105 is optimally close to glass 701.

FIG. 7a shows a conventional design of an adjustable mount. As one cansee, when camera 104 is adjusted for the sharp rake angle of windshield701 a camera 104 moves away from the glass. The same problem occurs if aball socket is used with the ball centered above or ball camera 104 orif camera 104 fits inside a sphere mounted inside a ball socket of thesame diameter so that camera 104 may be adjusted in all three angles.

Embodiment 1 Axis of Rotation Close to the Glass

A camera mount 70 is desirable that allows adjustment of the cameraangle but enforces a constraint that camera lens 105 touches glass 701.Reference is now made to FIGS. 7b and 7c which show an embodiment ofcamera mount 70 according to the present invention. In FIG. 7b , camera104 is shown butted up against windshield glass 701 shown at twodifferent angles. Camera mount 70 includes a fixed element 707 mountedflush with windshield 701. Reference is now also made to FIG. 7c whichillustrates the embodiment of camera mount 70 of FIG. 7b in more detail.An axis 703 of rotation has been shifted to the tip of lens 105 wherethe lens mount 105 touches the glass. The body of camera 104 has twoextensions 705 one on each side of lens 105. The tip of extension 705 isin line with upper edge of lens 105. From each extension 705, axis pin703 is attached which is parallel to the glass plane. Axis pins 703 fitinto corresponding holes in extension 705. If we wish to have lens 105actually touch the glass, the holes in the mount extensions 705 may bereplaced by grooves and the glass itself closes onto pin 703. After thecamera angle is adjusted, camera 104 is locked into position optionallyby applying an adhesive, e.g. epoxy adhesive, between extension 705 andfixed element 707. Alternatively, a set screw or any other method ascommonly known in the art may be used for locking.

FIG. 7d shows a modification of the previous design (FIGS. 7a-7c ) inwhich an arm 709 with a slot is attached to each side of fixed element707 and camera body 104. (only one side is shown) Arm 709 rotates arounda point 711 on fixed element 707 and slides along pins 713 protrudingfrom camera body 104. If pins 713 are threaded, a nut or wing nut may beused to lock the camera angle.

Further modifications of the design are possible. Reference is now madeto FIG. 7i , illustrating camera mount 70 according to anotherembodiment of the present invention. Arm 709 may have another hole, onefor the pin at point 711 connecting to fixed element 707 and the otherfor a pin 716 attached to camera body 104. The axis at point 703 at thetip of lens 105 may slide in a slot along the glass of windshield 701.If the pin at point 703 is not actually at the tip of lens 105 butfurther back, a curved slot may be designed that constrains the tip oflens 105 to move along a straight path while touching the glass.

Embodiment 2 Semicircle Design

Reference is now made to FIG. 7e which illustrates two views of a secondembodiment of the present invention. The two views differ by the anglesof windshield 701. As camera 104 is rotated so that the tip of lens 105touches glass 701 at point 703, the opposite corner 704 of camera body104 marks a circular path whose center is point 703 of contact betweenlens 105 and glass 701. Since opposite corner 704 is the most distantpoint on camera 104 from the tip of lens 105, the semi-circle shownencloses all of camera 104 during adjustment. Reference is now also madeto FIGS. 7f and 7g which show a side view and rear view respectively, ofcamera mount 70, according to an embodiment of the present invention.Camera body 104 may be modified so that instead of a sharp corner atpoint 704 it has a flat or beveled surface 704 at the point of contactwith a semi-circular strip 714. It should be noted that the flat surfaceat point 704 is perpendicular to a radius of semi-circular element 714.One can prove, using elementary geometry, that if a flat surface atpoint 704 is pulled tight towards the semi-circular element, the tip oflens 105 touches center 703 of semi-circular element 714 which is onglass 701, regardless of where on semi-circular element 714 the flatsurface touches. One way to pull the flat camera surface tightly towardsthe semi-circular mount 714 is with a fastener 717, e.g. wing nut orscrew. A slot, e.g. of two centimeters, in width is bent into asemi-circle and a slot 715 is cut into it. A threaded hole may be tappedthrough flat surface on camera body 104. Screw 717 is optionallyinserted through slot 715 and then screwed into camera body 104. Whenscrew 717 is tightened the flat surface is pulled tightly towardssemi-circular element 714.

Another option is to glue the flat surface to semicircular element 714once the correct angle has been determined. Another option is to havethe semicircle ferromagnetic and camera body 104 be a magnet. Camera 104is typically constrained to have only one degree of freedom in movementsince the flat surface is a rectangular shaped and the minimum distance(radius) is achieved when opposite edges of the rectangle touchsemi-circular element 714. If however, camera body 104 is part of asphere, then the surface at point 704 is a circle. Semi-circular element714 may be formed from a hemisphere. Screw 717 may then give a secondaxis of rotation and allow for an adjustment of camera roll. Somelateral rotation (pan) motion is also possible. Semi-circular element714 may be a portion of a sphere and camera body 104 may be a portion ofsphere of the same diameter, so that tip 703 of lens 105 is the centerof the sphere. The screw hole may be placed on any location on camerabody 104. The screw hole may be placed on the optical axis of camera 104enabling two axis of motion. Vertical motion to account for windshieldrake angle and rotation around the optical axis to account for roll. Ifcamera body 104 is attached to mount 714 using glue or a magnet then thehemisphere inside hemisphere design gives also a lateral rotation (pan)by relaxing the requirement that tip 703 of lens 105 touches glass 701

Reference is now made to FIG. 7h , illustrating a method for mounting acamera behind a windshield according to an embodiment of the presentinvention. A tilt rotation axis is provided (step 750) at a pointbetween the lens mount and the windshield. The front tip of the lensmount is constrained (step 752) to be in close proximity to or tocontact the inside of the windshield for different rake angles of thewindshield.

The indefinite articles “a”, “an” is used herein, such as “an imageframe”, “a traffic sign” have the meaning of “one or more” that is “oneor more image frames” or “one or more traffic signs”.

In the context of a patterned filter, “ . . . ” means that the patterncontinues to repeat.

Although selected embodiments of the present invention have been shownand described, it is to be understood the present invention is notlimited to the described embodiments. Instead, it is to be appreciatedthat changes may be made to these embodiments without departing from theprinciples and spirit of the invention, the scope of which is defined bythe claims and the equivalents thereof.

What is claimed is:
 1. A method for angular adjustment of a cameramountable inside a windshield of a vehicle, the method comprising:adjusting an angle of the camera around an axis at a front tip portionof the camera, by rotating a body of the camera while constraining, by acamera mount, the front tip portion to substantially contact an insidesurface of the windshield; and upon adjusting the angle of the camera,locking angular orientation of the camera.
 2. The method of claim 1,wherein the camera mount includes a fixed element, and the methodfurther comprises: mounting the fixed element on the inside surface ofthe windshield; and attaching a pin to the fixed element forconstraining the front tip portion of the camera to be in closeproximity to or to contact the inside surface of the windshield fordifferent rake angles of the windshield.
 3. The method of claim 1,wherein the camera mount includes a fixed element, and the methodfurther comprises mounting the fixed element on the inside surface ofthe windshield, the fixed element having a circular arc cross section ina vertical plane, wherein a tilt rotation axis is substantially at thecenter of the circular arc cross section.
 4. The method of claim 2,further comprising attaching the fixed element to a body of the camera.5. The method of claim 2, further comprising attaching an arm whichattaches the fixed element to the body of the camera.
 6. The method ofclaim 3, further comprising fastening a fastener which attaches througha slot in the fixed element having the circular arc cross section,wherein the fastener constrains the front tip portion of the camera tobe in close proximity to or to contact the inside surface of thewindshield for different rake angles of the windshield.
 7. A cameramount for angular adjustment of a camera mountable inside a windshieldof a vehicle, the camera mount comprising: an element configured to bemounted to an inside surface of the windshield; and an axis inconfigured to be attached to the element adjacent to the inside surface,the axis in being configured for mounting a front tip portion of thecamera such that the tip portion is constrained by the axis in and theelement to contact the inside surface during an adjustment of an angleof the camera.
 8. The camera mount of claim 7, wherein the element isconfigured to be mounted flush to the inside surface of the windshield.9. The camera mount of claim 7, wherein the axis pin is configured to beconnected to an extension structure of the camera.
 10. The camera mountof claim 7, further comprising an arm having a slot configured to beattached to the element and a camera body, wherein the arm is configuredto be rotatable around a second axis pin attached to the element at alocation different from a location of the axis pin, and wherein the armis slidable along a pin protruding from the camera body.
 11. The cameramount of claim 7, further comprising an arm attached to the element at afirst end and to a camera body at a second end, wherein the axis pin isslidable along a slot in the windshield.
 12. The camera mount of claim7, further comprising only one locking mechanism configured to lock theangle of the camera after the adjustment.